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The Value of Including Estimates of Precision 
in Finished Intelligence 


It has often been noted in RMS that resource management of the 
intelligence community is a-ganized around the technical collection systems. 
There is wide agreement that this is suboptimal and that instead we should 
Manage resources to accomplish our abjective, support to intelligence users. 
We do attempt to assess how well collection improvements serve this purpose ty 
estimating the increased intelligence information that may be obtained from 


improvements to the collection system, but no one pretends that this is optire:: 


since only those improvements in or support to users that can be obtained 
through improvements to the technical collectors are considered. Thous:: 
attempts have been made, it is not sufficient tc simply extend this empiasi. 2 
technical collection initiatives to all collection initiatives since the mo 
useful and cost-effective improvements ‘in support to users may not even 
discovered through this approach. Naor is it sufficient to ask how to ingrown 
processing, exploitation, dissemination, @& production for the same reason ~ 
there is no guarantee that all of the most promising opportunities far 
improving support will be considered because the focus of this approicn. .3 
means rather than ends. 


I have come to believe that the wincipal reason that the managemes! ~~ 
the Intelligence Camunity is collecta’ oriented is that there is inforsat bus 
to be had about the benefits of improving the allectors and, more Laie a 
there is little information available about how to improve use’ SUPpOrs- © % 
information that would appear at first glance to be mst desirable e ue 
improving intelligence comunity performance against 1ts objective is eps oi 
feedback from the decisionmaker about the usefulness of cur SUPPE~: = ii ‘ 
the value of intelligence in providing support is straightfarward in te 
abstract, but very difficult to actually accomplish — partly becals® ©" © 
is a composite of a number of individuals ratha than a unitary acto oe 
principally because the outcome measure is difficult to define in a meats 
way. 
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in Finished Intelligence ——— 


The usefulness of additional intelligence support, @ any added 
information far that matter, to a decisionmaker can be defined operationally as 
changing the assessment of 


fe) consequences: a changed assessment of the consequences to be 
expected fran selecting alternative courses of action (costs, 
benefits, and risks), 


fe) uncertainty: an increase @ decrease in the uncertainty surrounding 
the estimates of costs, risks, benefits anticipated from alternative 
courses of action (note that support is improved by a more accurate 
portrayal of uncertainty rather than by simply maximizing confidence 
by minimizing uncertainty — this may not be evident to the 
decisionmaker ), 


ra) alternatives: the addition a elimination of alternative courses of 
action from the feasible set, @ 


? ° timing: changes in the timing of the decision (finding that it can 
be made earlier oc later) a@ changes in the expected timing of the 
consequences of selecting an alternative (later a earlier arrival of 
costs, benefits, and risks). 


We have conducted user surveys to obtain this information but the results have 
been disappointing, primarily because there is no clear audit trail available 
to define the affect of intelligence information on a decision and also because 
there is no unitary actar to query fa’ informal recollection of these effects. 
It is difficult as well to identify intelligence shortfalls because there is a 
strong interaction between what can be done £7 the decisionmaker and what they 
in turn request a expect. Because the demana for intelligence is ephemeral 
and easily influenced by expections and changing events, it is difficult to 
distinguish highly useful intelligence from barely useful intelligence and 
close to impossible to make finer distinctions. Furthermore, to be usefully 
precise, demand must be expressed as a result af painstaking internal 
comparisons such as those made when operating under a budget. No such 
discipline is imposed on the intelligence user and this inevitably diminishes 
the value of user surveys. 


How then can we develop an integrated criterion for allocating resources 
in the community? Moving back from the user one step to the production of 
intelligence could be very useful if information were readily available about 
the difficulties in addressing intelligence issues that the shortages @ 
imprecision of intelligence information have caused. That information is 
currently not available. Judging fram the intelligence products I have 
examined over the past 2-1/2 years, the current production process disguises 
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that small amount of information about uncertainty that would acdinarily be 
evident to the reader by suppressing dicageements uncovered through 
coordination by resorting to vaguer language & by eliminating those aspects fora 
the issue that are in dispute from the final version. This is exacerbated by a 
style of prose that is highly readable but designed to disguise differences in 
certainty fram one topic to another by giving 4 uniform impression of certainty 
mitigated by the extensive use of qualifiers in the text and by including 
precise statements about highly conditional events. (This is especially 
misleading when the likelihood of the conditioning events is poorly known. ) 


An approach that would provide substantially more information on which to 
base resource decisions better would be to include formal assessments of the 
accuracy and precision in the final product using, where possible, statistica. 
measures of bias and variation. Where the methods of statistics and 

engineer ing could not be used (such as political analysis in som instances), 
it would be sufficient to descr ibe the range of possible values. In that case 
it would be necessary to include some discussion of where in that range the 
best estimate lies and why this is so- If there is no possibility of 
specifying upper and lower bounds that result in a usefully narrow range, a 
subjective 90% confidence interval could be specified. This is done by 
indicating the upper and lower values about the estimate that define the rance 
that the analyst expects will include the correct value 90% of the time. 


vA 
This approach would provide a wealth of information on which to base 
resource allocation decisions. it would also substantially improve support tO 
users in two ways: 


re) they would understand much better the uses and limits of the 
intelligence they receive amd, 


° an additional avenue would be provided to the intelligence analyst 
far communicating concern about events with poorly estimat 
probabilities that are important nonetheless for Gecisionmaking- 


An alternative implementation of this same approach would be to have the 
process of formally specifying the accuracy and precision of intelligence 
assessments assigned to a separate gcOuP of intelligence analysts. They would 


information as well as their ability © estimate the errors introduced by the 

estimating procedures through sensitivity testing and contingency analysis. #n 
advantage of establishing a separate goOUP for this function is that the Qoup 
could also be tasked to identify those applications most in need of improvement 
and to review the potential of resource allocation proposals far providing such 


improvement. 
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